feature: Add support for native PT DDP distribution #3223

vishwakaria · 2022-07-11T00:49:40Z

Issue #, if available:

Description of changes:

Allow customers to launch native PTDDP jobs with nccl backend using a new distribution value: { "pytorchddp": "enabled": "true", <other options> }}
Uses smdataparallel runner for launch, same as smdistributed backend.
Design document
Connected PRs: feature: add support for native PT DDP distribution sagemaker-pytorch-training-toolkit#236

Testing done:

https://quip-amazon.com/usIaAMvBbknO/Test-Plan-PT-DDP-Launcher

Note: This PR can only be merged after the sagemaker-pytorch-training-toolkit PR has been merged.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

[ X ] I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
[ X ] I used the commit message format described in CONTRIBUTING
[ N/A ] I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
[ X ] I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

[ X ] I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
[ X ] I have checked that my tests are not configured for a specific region or account (if appropriate)
[ N/A ] I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-07-11T00:53:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 38719d3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-11T01:25:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 38719d3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-11T01:38:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 38719d3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-11T01:57:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 38719d3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-11T03:42:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 38719d3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-11T22:59:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6b33897
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-11T23:42:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6b33897
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-11T23:43:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6b33897
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-12T00:01:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6b33897
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-12T01:50:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6b33897
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-15T22:20:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fedc1c2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-15T22:48:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: fedc1c2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-15T22:49:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: fedc1c2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-15T23:19:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: fedc1c2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-15T23:25:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fedc1c2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T00:09:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a4352c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T00:39:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a4352c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T00:52:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a4352c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:13:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a4352c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-16T01:21:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a4352c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-18T23:56:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 81ba403
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-18T23:58:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 81ba403
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-19T00:21:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5cce7a0
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-19T00:53:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 81ba403
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-19T01:01:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 81ba403
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-19T01:03:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 820433d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-07-19T01:06:11Z

Codecov Report

Merging #3223 (905b6b2) into master (38b4916) will decrease coverage by 0.41%.
The diff coverage is 97.43%.

@@            Coverage Diff             @@
##           master    #3223      +/-   ##
==========================================
- Coverage   89.40%   88.99%   -0.42%     
==========================================
  Files         848      203     -645     
  Lines       72563    18199   -54364     
==========================================
- Hits        64877    16196   -48681     
+ Misses       7686     2003    -5683

Impacted Files	Coverage Δ
src/sagemaker/inputs.py	`100.00% <ø> (ø)`
src/sagemaker/jumpstart/types.py	`94.37% <0.00%> (ø)`
src/sagemaker/mxnet/model.py	`98.18% <ø> (+1.81%)`	⬆️
src/sagemaker/pytorch/model.py	`96.36% <ø> (+10.90%)`	⬆️
src/sagemaker/sklearn/model.py	`93.87% <ø> (+12.24%)`	⬆️
src/sagemaker/workflow/step_collections.py	`93.87% <ø> (ø)`
src/sagemaker/job.py	`90.00% <75.00%> (-0.55%)`	⬇️
src/sagemaker/chainer/model.py	`95.91% <87.50%> (-1.65%)`	⬇️
src/sagemaker/jumpstart/cache.py	`95.68% <93.54%> (-0.90%)`	⬇️
src/sagemaker/utils.py	`92.72% <95.23%> (+0.03%)`	⬆️
... and 686 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4afe7f...905b6b2. Read the comment docs.

sagemaker-bot · 2022-07-19T01:06:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5cce7a0
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-19T01:20:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 820433d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

roywei

overall LGTM, few comments.

roywei · 2022-07-28T22:42:56Z

src/sagemaker/fw_utils.py

@@ -103,6 +103,17 @@
        "1.11.0",
    ],
 }
+
+PYTORCHDDP_SUPPORTED_FRAMEWORK_VERSIONS = [
+    "1.10",


I don't see 1.10.2 here. Also are these versions essentially use the same DLC docker? which is 1.10.2? as we only have one DLC docker file for each minor release.

Adding 1.10.2.
Yeah, seems like there is only one DLC docker file for 1.10 (which is for 1.10.2): https://github.com/aws/deep-learning-containers/blob/master/available_images.md#prior-sagemaker-framework-container-versions

roywei · 2022-07-28T22:50:55Z

src/sagemaker/fw_utils.py

+        return
+
+    err_msg = ""
+    if not image_uri:


what will happen if using an custom docker image uri and there is old toolkit installed there?

I am mirroring the logic that we have for smdistributed: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/fw_utils.py#L635.
How can we check for older toolkit versions inside custom DLCs?

roywei · 2022-07-28T22:51:33Z

tests/conftest.py

+
+
+@pytest.fixture(
+    scope="module", params=["1.10", "1.10.0", "1.10.1", "1.11", "1.11.0", "1.12", "1.12.0"]


1.10.2 is missing, also is 1.10.1 a valid version that we released? we may have skipped it

Gotcha, I will switch to 1.10.2 then.

sagemaker-bot · 2022-07-28T23:03:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a013b71
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T23:21:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a013b71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T23:25:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a013b71
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T23:48:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a013b71
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T23:51:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a013b71
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T00:32:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 905b6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T00:42:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 905b6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T00:53:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 905b6b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T01:15:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 905b6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T01:23:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 905b6b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T05:05:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 905b6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T05:35:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 905b6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vishwakaria · 2022-07-29T19:05:47Z

Moved to #3270 after rebase and squashing commits.

vishwakaria force-pushed the ptddp-launcher branch from 81ba403 to 5cce7a0 Compare July 19, 2022 00:16

vishwakaria and others added 11 commits July 28, 2022 22:32

Fix Flake8 and black-check errors

9dcbb02

ptddp doc update

458031d

move the location of the note

5d0bef2

minor fixes, reduce num_instance to two

86f9667

minor doc improvements

72bb2d8

minor fixes

46f882c

minor doc improv

a21719e

fix indentation

6c28d98

Retrigger CI checks

a8016c0

Add validation for py and framework version supported

77848f2

Minor fix unit tests

a013b71

vishwakaria force-pushed the ptddp-launcher branch from 2c5212e to a013b71 Compare July 28, 2022 22:35

roywei reviewed Jul 28, 2022

View reviewed changes

Add 1.10.2 to PT supported versions

905b6b2

vishwakaria closed this Jul 29, 2022

vishwakaria mentioned this pull request Jul 29, 2022

feature: Add PyTorch DDP distribution support #3270

Merged

2 tasks

vishwakaria deleted the ptddp-launcher branch August 16, 2022 23:46



		@pytest.fixture(
		scope="module", params=["1.10", "1.10.0", "1.10.1", "1.11", "1.11.0", "1.12", "1.12.0"]

feature: Add support for native PT DDP distribution #3223

feature: Add support for native PT DDP distribution #3223

Conversation

vishwakaria commented Jul 11, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 11, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 19, 2022

AWS CodeBuild CI Report

codecov-commenter commented Jul 19, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Jul 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 19, 2022

AWS CodeBuild CI Report

roywei left a comment

Choose a reason for hiding this comment

roywei Jul 28, 2022

Choose a reason for hiding this comment

vishwakaria Jul 29, 2022

Choose a reason for hiding this comment

roywei Jul 28, 2022

Choose a reason for hiding this comment

vishwakaria Jul 28, 2022

Choose a reason for hiding this comment

roywei Jul 28, 2022

Choose a reason for hiding this comment

vishwakaria Jul 28, 2022

Choose a reason for hiding this comment

sagemaker-bot commented Jul 28, 2022

vishwakaria commented Jul 11, 2022 •

edited

Loading

codecov-commenter commented Jul 19, 2022 •

edited

Loading